The present disclosure relates to an image processing apparatus for producing an image with a wide dynamic range by making use of a plurality of images taken at different exposure times, an image taking apparatus employing the image processing apparatus, an image processing method for the image processing apparatus and an image processing program implementing the image processing method.
More particularly, the present disclosure relates to a mechanism for suppressing flickers generated in a video and flickers of a color-signal system in an operation to generate an image with a wide dynamic range by making use of a plurality of images having different exposure times. These flickers are caused by differences between optical sources existing for taken images, fluctuations included in illuminated light as fluctuations of a power-supply frequency or characteristics of the afterglow. The flickers generated in a video are also referred to as optical-source flickers whereas the flickers of a color-signal system are also referred to as color flickers or color rolling.
An image taking apparatus for taking an image of an image taking object includes a mechanism for controlling the quantity of light incident to a pixel section of the apparatus. The image taking apparatus is typically a solid-state image taking apparatus of a CCD (Charge Couple Device), MOS (Metal Oxide Semiconductor) or CMOS (Complementary MOS) type. In the following description, the pixel section is also referred to as an image taking section, a pixel array section, an image sensor or an opto-electrical conversion sensor whereas the mechanism is referred to as an incident-light quantity controller.
The incident-light quantity controller is a controller having a mechanical diaphragm mechanism provided on an image taking lens thereof or a controller having a mechanical shutter mechanism provided on an image taking lens thereof. In the following description, the mechanical diaphragm mechanism is referred to as a mechanical iris whereas the mechanical shutter mechanism is referred to merely as a mechanical shutter. As an alternative, the incident-light quantity controller can also be a controller having the so-called electronic shutter function capable of controlling the length of an accumulation time of signal electrical charge in the pixel section of the solid-state image taking apparatus. In the following description, the accumulation time is also referred to as an exposure time.
The mechanical iris and the mechanical shutter can be used independently of each other. However, the mechanical iris can, also be used by combining the mechanical iris with the mechanical shutter or the electronic shutter.
By the way, a variety of image taking apparatus do not raise a problem if the apparatus are used for taking an image by making use of an optical source put in a steady state in which the brightness of light generated by the source does not change. If an image is taken by making use of an optical source such as a fluorescent lamp having a periodical light emitting characteristic and operating asynchronously with the exposure period of the semiconductor image taking apparatus, however, optical source flickers are generated.
It is to be noted that the optical source flickers are distinguished from luminance flickers which are flickers of screen luminance and from color reproduction flickers which are also referred to simply as color flickers or color rolling.
The optical source flickers are perceived as a phenomenon in which a video signal changes due to a relation between luminance changes of an optical source and the exposure period of an image taking apparatus.
For example, the luminance signal component of a video signal changes due to luminance changes of an optical source making use of a commercial power supply having a frequency f and due to beat components having a field period fv of the image taking apparatus. In this case, the luminance changes are changes within a period of 1/nf seconds where reference symbol n is normally the integer 2. Since the luminance signal component of a video signal changes, the output video also varies as well at a period also related to the afterglow characteristic of the eye of a human being. A phenomenon in which image flickers are felt is referred to as luminance flickers.
For example, luminance flickers are generated with ease in an area in which the NTSC system having a field frequency of 60 Hz is adopted and the frequency f of the commercial power supply is 50 Hz. Luminance flickers are also generated with ease in an area in which the PAL system having a field frequency of 50 Hz is adopted and the frequency f of the commercial power supply is 60 Hz. In addition, in comparison with the electrical light bulb, the luminance of the fluorescent lamp changes due to the light emitting characteristic of the fluorescent lamp so that luminance flickers are generated very considerably by the fluorescent lamp.
It is to be noted that a statement saying a field frequency of 60 Hz can be said in other words as a statement saying a frame frequency of 30 fps. Speaking more accurately, the field frequency is 59.94 Hz. On the other hand, a statement saying a field frequency of 50 Hz can be said in other words as a statement saying a frame frequency of 25 fps.
For example, the emission period of the fluorescent lamp is 10 ms whereas the period of the exposure operation in the NTSC system having a field frequency of 60 Hz is 16.7 ms. In this case, the lowest common multiple of the emission period of the fluorescent lamp and the period of the exposure operation in the NTSC system is 50 ms. That is to say, in three exposure operations, the relation between the emission period of the fluorescent lamp and the period of the exposure operation in the NTSC system is restored. Thus, there are three kinds of exposure period. Differences of the levels of signals output by the solid-state image taking apparatus between these three exposure periods cause flickers to be generated at a flicker frequency F of 20 Hz.
In addition, if the function of the electronic shutter is used, the higher the speed of the shutter operating in the shutter mode, the shorter the time included in the one field period as an accumulation time for accumulating electric charge in the solid-state image taking apparatus.
Thus, the amplitude of flickers becomes larger than that for a normal shutter speed of 1/60 seconds. The higher the speed of the electronic shutter, the more striking the generated flickers. As a result, flickers including mainly image luminance flickers appear on the screen, causing the quality of the image displayed on the screen to deteriorate considerably.
In addition, the green, red and blue colors are three colors of a fluorescent substance used in a fluorescent lamp. Even though the emissions of the three colors start with the same timing, the light quantities of the three colors decrease at different rates so that the three colors disappear eventually at different times. That is to say, the light emitted by the fluorescent lamp changes its spectrum with the lapse of time.
In general, the emission time of the green color is particularly longest among the three colors. On the other hand, the emission time of the blue color is shortest among the three colors. That is to say, the emission time of the red color is between those of the green and blue colors.
Thus, depending on the shutter timing of the shutter having a high speed, only one or two color components of the emitted light can be taken in some cases.
In an operation to take an image by making use of an electronic shutter having a high speed, a difference in taken spectrum appears a color change. As described above, color reproduction flickers are also referred to as color flickers or color rolling.
In particular, the light component of the blue color cannot be taken as color flickers. In many cases, if an image taking operation is carried out normally, the light component of the blue color is taken inadvertently as the light component of the yellow color.